feat: simpatico zonemaps (chunk pruning) - #1736

…phase the work

Plan audit: chunk_min is a value-domain minimum only for `input -> bitpack`
roots. `delta -> bitpack` (l_orderkey, o_orderkey) yields a min of deltas and
`str_split -> bitpack(offsets)` (l_shipmode) a min of string lengths, both
useless as zone maps; `dictionary -> bitpack` is usable because cuDF dictionary
keys are sorted. The free metadata is also contingent on a plan chosen for ratio,
and clustering a date column plausibly flips it to delta — destroying the free
min on exactly the column meant to do the pruning. So min/max is stored
out-of-band, unconditionally, and the in-payload arrays become a second-level
in-kernel early-out.

Granularity: one entry per group of G simpatico chunks, G configurable, default
8 (8192 rows). Indistinguishable from 1024 in pruning power on every layout
measured, 8x cheaper (0.17 GB vs 1.36 GB for SF1000 lineitem), and G | 120 tiles
the 122,880-row pin unit exactly. Adds the plan doc: at the configured 8 GB batch size
a pin chunk is 42% of customer and 81% of part, so pin-chunk granularity is not
a viable mechanism for most tables.

Work order rewritten as a staged proof of value: the clustering experiment builds a clustered
SF100 and re-runs the explorer; the pruning sweep is zone-map capture on the compressed pin path plumbing plus a scan_task_batch_size
sweep read as an on/off delta, which answers the whole granularity question in
real query time without building the index; the group-index work builds it only if the pruning sweep pays.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…works, and it is cheap on the GPU

Built /datasets/tpch_sf100_sorted (lineitem by l_shipdate, orders by o_orderdate).
Row-group spans collapse 2525 -> 4.4 days and row-group pruning goes from 0.00%
to 36.45% of rows, inside the 34-38% band predicted from the SF10 sweep.

The delta-flip risk is refuted: the explorer keeps `input -> bitpack` on every
clustered column, with l_shipdate's ratio jumping 2.651x -> 426x. Inside a
1024-row chunk of clustered data the values are near-identical, so chunk_bits
collapses and plain bitpack already wins. Clustering strengthens the free
chunk_min rather than destroying it.

New the plan doc on clustering cost and strategy:
- GPU sort is ~34x faster than 72 CPU threads (2659 vs 79 Mrows/s). A per-pin-chunk
  local sort of SF1000 lineitem costs ~2.3 s, ~1.5% of the existing ~151 s pin.
  Cost is the gather, not the comparison, so it scales with pinned bytes and can
  in principle fuse into the materialize->compress path.
- A global sort is not required. Local sort inside each pin chunk reaches 72.7%
  against a global sort's 73.5% at G=8 -- but 0.0% at pin-chunk granularity.
  Range-partitioning gets 69.0% coarse but needs a full-table shuffle.
  Local sort + a G=8 index ~= a global sort with no shuffle: the strongest
  argument yet for the fine index.

Methodology trap recorded: DuckDB's FILE_SIZE_BYTES rotation does not preserve a
global ORDER BY across files; the first dataset build was silently wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…rovenance

Two corrections to the clustering experiment.

1. "String columns carry no min/max statistics at all" was wrong -- verified 0 of
   94 row groups missing them at both SF100 and SF1000. Min/max fails on those
   columns for a distributional reason (a 1M-row group of a 3-to-7-value column
   holds every value), not a missing-statistics one. the analysis's argument for
   present-value bitmaps stands, but the stated reason is now the right one:
   parquet's answer to set membership is the bloom filter, which these files
   lack, not the min/max that they have.

2. The clustered tree is written by DuckDB 1.5.5 while every other /datasets/
   tree comes from parquet-rs 57.3.1, and the encodings differ substantially
   (DELTA_BINARY_PACKED vs PLAIN, RLE_DICTIONARY/uncompressed vs
   PLAIN_DICTIONARY/snappy). Decimals stay INT64-backed on both so pushdown is
   unaffected, but absolute scan times are not comparable across the trees.
   Adds /datasets/tpch_sf100_duckdb_natural -- same tables, same writer, no
   ORDER BY -- so a sorted-vs-unsorted timing comparison isolates row order.
   the pruning sweep's primary experiment (pruning on vs off on one dataset) is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The device pin path forced capture_chunk_stats off because device_pin_result had
nowhere to put the statistics, so the configuration that wins the suite by
17.8-19.7% (GPU-tier compressed) was the one configuration with no zone maps at
all. Everything downstream already existed: compute_pinned_chunk_stats runs on the
uncompressed GPU table before compression, pinned_zone_maps normalizes the capture,
and build_cached_scan_plan / chunk_provably_empty already prune with it.

Four plumbing changes, mirroring the host path exactly:
- device_pin_result gains chunk_stats
- materialize_all_batches_compressed stops forcing capture off and collects them
- insert_pinned_entry_device takes column_types + chunk_stats and builds the sidecar,
  including the same shape-mismatch warning the host path emits
- the extension passes capture_chunk_stats through and hands the stats to the insert

Verified on the clustered SF100 (/datasets/tpch_sf100_sorted) with all eight tables
pinned GPU-tier compressed: q6 prunes 26/34 chunks, q14 prunes 28/30 -- the first
end-to-end chunk skipping on a compressed pin. Unit tests pass (189 cases).

Refs CHUNK_SKIPPING_PLAN.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ath landing and the first end-to-end prune

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… the compressed pin path is worth ~4%, and granularity needs an index

Sweep on clustered SF100, all eight tables GPU-tier compressed, best-of-3:

  batch    pruned   ON        OFF       ON-OFF
  8GB      25%      0.9551 s  0.9544 s  +0.1%
  2GB      48%      0.9348 s  0.9493 s  -1.5%
  512MB    66%      1.0292 s  1.0720 s  -4.0%
  128MB    71%      1.5801 s  1.8085 s  -12.6%

22/22 byte-exact against DuckDB CPU with pruning on.

Pruning benefit grows monotonically as granularity refines, and the prune rate
approaches the 73.5% ceiling the plan doc predicted. Since pruning is governed by a
chunk's fraction of the table, SF100@512MB (1.8-4.0%) is the proxy for SF1000@8GB
(3.1%) -- so zone-map capture alone is worth ~4% of suite time on clustered data.

The sweep also shows you cannot buy granularity with batch size: 128MB prunes the
most yet is the slowest arm, because batching overhead outgrows the saving. That
is the argument for the group-index work -- granularity has to come from an index. But on
SF1000 lineitem the index only adds ~7 points of prune rate; its real value is the
small tables (a pin chunk is 42% of customer, 81% of part) and enabling the cheap
local-sort clustering that yields 72.7% fine and 0.0% coarse.

Also records the process failure: the first sweep run was silently invalid because
a $( ... && ... ) inlined into a sed replacement broke the shell AND operator and
set both arms to false. The script now verifies the injection and fails if a
prune-on arm logs no pruning.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Storage was never the worry; compute might have been. cudf::segmented_reduce over
a fixed-stride offsets column, 189M-row chunk, all 16 lineitem columns, GB300:

  G=1  (1024 rows)  184,571 groups  0.0209 s  5.8x
  G=8  (8192 rows)   23,072 groups  0.0061 s  1.7x
  G=64 (65536)        2,884 groups  0.0061 s  1.7x

against the 0.0036 s of whole-chunk cudf::minmax that zone-map capture on the compressed pin path already pays. Building the
whole G=8 index for SF1000 lineitem is ~0.20 s against a ~151 s pin (0.13%).

Third independent argument converging on G=8: G=1 buys no extra pruning, costs 8x
the storage, and costs 3.4x the compute. Coarser than 8 buys nothing either --
G=64 is the same 0.0061 s because the reduction is already bandwidth-bound.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… plan, and the three serve routes

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three independent processes on the 8GB and 2GB arms. Suite totals spread ~0.8%
run-to-run, so the 8GB delta (+0.1/-0.0/+0.4%) is noise while the 2GB delta
(-1.5/-1.5/-2.4%) is not.

Falls out of that: scan_task_batch_size was tuned to 8GB on unclustered data with
no pruning, where bigger was strictly better (the yaml records 5GB->8GB as -1.85%).
With zone-map capture on the compressed pin path on clustered data the trade reverses at the margin -- 2GB-ON beats 8GB-OFF
by 1.8% reproducibly, with no code beyond zone-map capture on the compressed pin path. Worth re-testing at SF1000, where
8GB also peaks at 253.9 GB of 256 GB HBM.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

compute_pinned_group_stats computes min/max at G*1024-row granularity instead of
one cell per pin chunk, using one cudf::segmented_reduce per column rather than
one reduction per group. Same type allowlist, same "null cell never prunes"
contract, and the same null precision as the coarse capture (a column-level fact,
not a per-group count -- nothing consumes per-group null counts yet and they would
cost a second segmented reduction).

Measured at 1.7x the whole-chunk capture for a 189M-row chunk at G=8, i.e. ~0.20 s
to index all of SF1000 lineitem against a ~151 s pin (CHUNK_SKIPPING_PLAN.md). A group is a whole number of simpatico 1024-row decode chunks by
construction, so group g maps to decode chunks [g*G, (g+1)*G) with a shift.

Not yet wired to a consumer: the packed representation (surviving-group plan) and 2c (serving a
partial chunk) follow. Landing the capture separately keeps it testable in
isolation, which is where the edge cases are -- short final group, all-null group,
partly-null group, off-allowlist type, group_rows == 0, shape mismatch.

Tests: 25 cases / 243 assertions in [pinned_chunk_stats]. Full suite has the same
6 failing assertions as the dev build at 5201d9c (ORDER BY / TOP-N NULL placement
and two order-dependent flakes) -- no new failures.

Refs CHUNK_SKIPPING_PLAN.md. the per-group capture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…s does not scale

Adds a timing probe (tagged [.], run explicitly) for the the packed representation design question,
and records what it found:

  capture incl. host BaseStatistics construction  ~147 ns / group cell
  chunk_provably_empty (the plan-time probe)        83 ns / group cell

Irrelevant at 32 chunks (2.7 us total). At SF1000 lineitem with G=8 there are
732,000 groups, so one filter column costs ~61 ms of plan time per query and a
three-column predicate ~180 ms, against a 5.8 s suite.

So 2b must not store one BaseStatistics per (group, column) the way 2a's
chunk_group_stats does. It needs typed parallel min/max arrays evaluated
vectorized or on the GPU, with the TableFilter lowered once per query instead of
re-dispatched per cell. The 2a capture stays useful -- it is correct, tested and
GPU-side cheap -- but its output type has to change before anything consumes it.

Also revises the plan doc: device residency for the index is right, but the load-bearing
reason is evaluation throughput, not avoiding an H2D round trip.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…fetch

New the plan doc on whether chunk skipping can save the transfer, not just the decode.

It can, exactly. bp_offsets[c] is an exclusive scan of
(chunk_count[c]*chunk_bits[c]+31)>>5 (offsets_cumsum.cu:9-11), so a 1024-row
chunk's byte range inside a packed buffer is computable from two arrays totalling
5 B/chunk -- ~923 KB against ~8 GB of payload for a 189M-row chunk, 0.01%. And
bp_offsets is not even persisted; it is synthesized on device from stored data.

Per plan root: input->bitpack and delta->bitpack are both exactly addressable
(delta_first[c] is a per-chunk anchor, not a running global prefix, so chunks
decode independently); dictionary indices likewise; str_split needs a dependent
two-step read for chars; ans/snappy/lz4/bitcomp are opaque.

The plumbing already has the right shape: the host->GPU path fetches through
simpatico::payload_fetch_fn(offset, size, dst, stream) and already does
column-granular partial fetch. Byte ranges within a buffer is the same mechanism
one level finer.

Caveats recorded rather than assumed: decode guard words need slop in every range;
many small transfers only beat one big one if surviving groups coalesce into runs,
which is likely on clustered data but unmeasured and needs a fallback.

On a simpatico ingestion format: the addressability argument beats parquet's
(whose PageIndex neither Sirius nor DuckDB reads), but measure the host tier
first -- identical mechanism, no new format, and a clear target in turning
host-tier compressed from -7.0% into a win.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ormat, incl. the S3 case

Promotes the ingestion argument out of the plan doc into its own section.

What it buys over parquet: parquet's only skip granularity here is a ~1M-row row
group, since PageIndex/ColumnIndex are written by parquet-rs but never read by
either Sirius or DuckDB. A simpatico file carries exact 1024-row addressability
for free from chunk_count/chunk_bits -- three orders of magnitude finer.

The S3 case, where the gap is widest: parquet's row-group size fights the storage
layer's own striping. S3 stripes and replicates internally with a layout we
neither control nor see, so a ranged GET inside a ~1M-row row group still makes
the backend reconstruct whatever internal unit it lands in -- a "pruned" parquet
read plausibly costs the backend as much as an unpruned one. Finer granularity
decouples what we skip from what the backend must restore. Flagged explicitly as
an unmeasured hypothesis about backend behaviour, and the interesting thing to
experiment with.

The tension: fine skipping is worthless over a network if one 8 MB GET becomes
two hundred 40 KB GETs. So read coalescing has to be first-class. The mechanism
already exists and is already per-backend -- io_context::align_and_coalesce, with
the REST variant taking a caller-supplied alignment as a pure coalescing knob
rather than a physical constraint. What is missing is policy, not machinery.

Recommendation: build it, but behind the host-tier fetch experiment. If range-
skipped fetch does not pay over a 370 GB/s C2C link where round-trips are nearly
free, it will not pay over S3 where they are not. And settle the S3 striping
question with a standalone bucket probe before any format work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ed from bitpack

Correction. the plan doc derived group byte ranges from bp_offsets, i.e. from bitpack's
chunk_count/chunk_bits channels -- reintroducing exactly the compression-plan
dependency that the plan doc/the in-payload bitpack metadata exists to avoid, one layer below the min/max index. The
explorer optimises ratio and decode throughput and nothing keeps a column on a
bitpack root, so a plan change would silently remove the addressing.

The mechanism is instead a second out-of-band table, emitted by the encoder: group
id -> byte range per bulk leaf buffer, at the same G=8 granularity as the min/max
index (they must match -- we skip at group granularity, so ranges are needed at
group granularity). 8 B per group per bulk buffer, ~3.7 MB against ~8 GB of
payload for 16 lineitem columns, 0.046%. Larger than the 5 B/chunk bitpack
derivation and worth it to be plan-independent. An operator that cannot be
group-addressed emits no table and its buffers are fetched whole -- graceful
degradation, exactly like a missing zone-map cell never prunes.

Keeps the per-operator analysis but reframes it as "can emit a table" rather than
"is derivable", and notes the consequence: fetch-skippability becomes a third axis
on plan choice that the explorer does not currently see.

Fixes the same claim in the plan doc and the persisted-index text in the plan doc

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Same experiment as run-sweep.sh at production scale, on the clustered SF1000
dataset, reusing the tuned sf1000-repro config so the only variables are
scan_task_batch_size and enable_pinned_zone_map_pruning.

Repoints downgrade_root_dirs, which the base config aims at a /localhome path
that does not exist on this box. Carries the same config-injection guards as the
SF100 sweep: compute the flag before the sed, verify both injections, and fail if
a prune-on arm logs no pruning.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ep projection

Built and verified a globally sorted SF1000 (row counts and checksums match
source; lineitem row-group spans 2525 -> 0.44 days) and ran the on/off sweep with
the tuned sf1000-repro config.

  batch  pruned  ON        OFF       ON-OFF
  8GB    17%     6.9625 s  6.9903 s  -0.4%
  2GB    45%     7.2858 s  7.4372 s  -2.0%

The ~4% projected from SF100 did not materialise; the real number at the
production 8GB batch is -0.4%, at or below the noise floor.

Root cause, and it dominates the whole project: pin chunks are not clustered even
when the file is. A single-day predicate on the globally sorted table prunes only
13/37 chunks -- 24 of 37 pin chunks contain one particular day. Scan thread count
(18 vs 3) and lexicographic file ordering were both tested and eliminated. The
cause is documented in the pin path itself, src/pin_table.cpp:260-262: "chunk
ranges overlap, because the coalescer interleaves row groups rather than
partitioning the key space".

Consequences: clustering must happen at pin time, not in the dataset, which makes
the analysis's local-sort-per-pin-batch the only strategy that works rather than merely
the cheapest. Every the plan doc pruning number is an upper bound the current pin path
cannot reach. the group-index work is not invalidated and is arguably strengthened, but its
open question is now whether G=8 groups are fine enough to see through
row-group-granularity interleaving. And the SF100 result was lucky, not wrong --
six files and 512 MB batches rarely spanned much of the key space.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… not scan-bound

Two follow-ups that reframe the project.

(a) A fine index sees through the coalescer interleaving without any pin-time
sorting. The interleaving is at row-group granularity: sorted SF1000 lineitem row
groups are 1,048,576 rows, so a G=8 group (8,192 rows) sits entirely inside one
and inherits its 0.44-day span. Measured at row-group granularity on the sorted
SF1000, 36.54% of scanned rows are prunable (0.00% unsorted) against the 17% of
chunks zone-map capture on the compressed pin path prunes today. So the analysis's "clustering must happen at pin time" holds for
chunk-level pruning but is NOT a prerequisite for the group-index work -- the fine index is an
alternative to pin-time sorting, not a complement.

(b) But the ceiling is low. q6 -- filter and aggregate on lineitem, no joins, the
most scan-dominated query in TPC-H -- is 0.0413 s of a 6.963 s suite (0.6%). The
suite is q18 (24.8%), q9 (13.5%), q21 (12.3%). Extrapolating the measured deltas
to a hypothetical 100% prune rate gives about -2.4% at 8GB and -4.8% at 2GB. That
is the entire envelope on a GPU-resident compressed pin, and the group-index work's realistic
17% -> 36% is worth roughly -0.9%.

The "34-38% of scanned bytes" headline does not translate because on a GPU-tier
pin the payload is already resident and decodes at 500-1700 GB/s -- skipped bytes
are cheap bytes. Scanned bytes are the right metric for a fetch-bound tier.

So the mechanism works and is nearly free, but its value is concentrated in host
tier pins (where compressed host-tier loses 7.0% today entirely on fetch),
spilling configurations, and scan-heavy workloads. Measuring the host tier is the
experiment that decides whether the project has a target at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Same clustered SF1000, --pin host instead of --pin gpu. This is the tier where a
skipped chunk saves the payload H2D transfer, not just the decode.

  batch  pruned  ON         OFF        ON-OFF
  8GB    23%     11.0396 s  11.2354 s  -1.7%
  2GB    50%      9.6778 s  10.4428 s  -7.3%

Against -2.0%/-0.4% for the same arms on the GPU tier, and all far outside the
~0.8% noise floor. The scan-bound queries move most (q12 -0.124 s, q6 -0.101 s).

Confirms the the plan doc fetch hypothesis empirically before any implementation: q6 costs
0.0413 s on a GPU pin and 0.1143 s on a host pin, and that 2.8x is payload H2D --
exactly what skipping eats into.

The batch-size conclusion inverts on this tier. On GPU, smaller batches always
lost absolutely; on host, 2GB-ON (9.678 s) is the best configuration measured,
beating 8GB-OFF by 13.9%. Fetch granularity outweighs batching overhead once the
transfer is on the critical path.

the group-index work now has a quantified target at the production batch size: 8GB prunes 23%
for -1.7%, the analysis says a G=8 index reaches ~36% without pin-time sorting, and the
2GB evidence suggests a steeper-than-linear relationship, so -3% to -5% on host.

Not conflating tiers: host-tier compressed is still slower absolutely (9.68 vs
6.96 s). The claim is that chunk skipping recovers a meaningful part of the fetch
penalty that makes host tier lose, which is what matters when data does not fit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… guard the GPU

Second independent process plus a 512MB arm:

  batch   pruned  ON                  OFF                  ON-OFF
  8GB     23%     11.0396 s           11.2354 s            -1.7%
  2GB     50%      9.6250 / 9.6778 s  10.4733 / 10.4428 s  -8.1% / -7.3%
  512MB   69%     10.4173 s           11.9232 s            -12.6%

The 2GB arm reproduces across two processes, so the host-tier win is solid. The
curve has the same shape as the GPU tier, shifted: 512MB prunes far more (69%)
and shows the biggest delta, yet is slower absolutely because batching overhead
overtakes the saving. 2GB-ON at 9.625 s is the optimum.

That sharpens the group-index work's target to the same argument as the pruning sweep -- you cannot buy
granularity with batch size, so the index's job is to deliver 512MB's prune rate
at 2GB's batching cost. Scaling the 2GB saving to a 69% rate predicts ~9.3 s
against today's best 9.625 s, a further ~-3%.

Also adds require_idle_gpu to the sweep. bench-lock.sh serialises our own runs,
but a test binary started outside it in another worktree still contends: one run
here died at pool init with "failed to allocate 254818874163 bytes" while another
worktree's test_operator_sweep held 4.7 GB and 99% util. A contended run either
OOMs or silently reports inflated times, so the sweep now refuses to measure
until the GPU is under 2 GiB and 20% util, rechecked before every arm.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The group index cannot use one duckdb::BaseStatistics per (group, column):
chunk_provably_empty costs ~80 ns/cell (a stats Copy plus a virtual
CheckStatistics), and SF1000 lineitem at G=8 has ~732k groups -- ~61 ms of plan
time per filter column per query against a 5.8 s suite.

Adds packed_column_bounds (parallel min/max/valid arrays; every supported type
fits in 8 bytes, unsigned stored as bit pattern and compared via is_unsigned) and
lowered_bound_filter, which lowers a TableFilter once into a flat node array and
evaluates it with bounds arithmetic.

Measured on the same cells: 79.9 -> 3.4 ns/cell, 23x. That is ~2.5 ms per filter
column for SF1000 lineitem instead of ~61 ms.

Correctness is the whole risk here, since this replaces DuckDB's CheckStatistics
on the path that decides whether to drop data. Two defences:
- lower() gates on filter_safe_for_stats, the same allowlist the BaseStatistics
  path uses, so the two can never disagree about WHICH filters are evaluable.
- a cross-check test runs both evaluators over 336 (filter, range) pairs -- every
  admitted shape including nested AND/OR/OPTIONAL and IN, against ranges that
  straddle each constant, with and without nulls -- and requires zero
  disagreements.

Not yet wired to a consumer; 2c (serving a partial chunk) follows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…nal shorthand

Two cleanups.

Removes ~260 MB across 1381 files of harness output that a `git add -A` swept in:
per-query result dumps (q20 alone is 5.6 MB per run), Sirius logs and generated
configs. All of it is regenerable by re-running the sweep. The per-query runtime
CSVs are the part worth keeping, so those move to runtimes/ with a README saying
what each run was, and results*/ is now gitignored.

Also rewrites comments that referenced this work's internal shorthand -- planning
phase numbers and section numbers in CHUNK_SKIPPING_PLAN.md. Those mean nothing to
a reader outside the effort that produced them, and section numbers go stale as
soon as the doc is edited, so each comment now states the fact it was pointing at.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Zone-map pruning is all-or-nothing per pinned chunk today: a chunk whose min/max
overlaps the filter at all is served whole, even when most of its rows provably
cannot match. This adds the plan-side half of sub-chunk pruning.

pinned_entry gains group_bounds (per chunk, per cached column, min/max over
fixed-size row groups) alongside the existing coarse zone_maps, kept separate so
that sidecar's merge and degradation invariants are untouched. cached_scan_plan
gains survivor_row_ranges: the coalesced row ranges of each surviving chunk that
the per-group bounds cannot rule out.

Properties the rest of the system relies on, all covered by tests:
- Refinement only narrows an already-surviving chunk. It cannot resurrect a
  pruned chunk, and cannot drop a chunk entirely -- a chunk whose every group
  prunes would already have failed the coarse pass, since the chunk's own bounds
  contain every group's.
- Ranges coalesce, so a chunk with nothing prunable yields exactly one range
  covering it and costs a consumer nothing over serving it whole.
- An empty survivor_row_ranges means "serve whole chunks", so an entry without
  group bounds behaves exactly as before, and a consumer that cannot serve a
  partial chunk may ignore the field entirely -- always sound, just less
  selective.
- An absent group cell keeps its rows, mirroring a null BaseStatistics.
- The all-pruned sentinel chunk is left unrefined, so narrowing cannot empty it
  and reintroduce the zero-batch hazard that stalls pipeline completion.

Filters are lowered once per plan rather than per group cell, which is what makes
a ~700k-group index affordable to evaluate.

No consumer reads survivor_row_ranges yet; the serving paths still take whole
chunks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…nnable alone

The index is only useful if it can be read without reading the data it describes;
that ordering is the whole premise of skipping a fetch. Neither current layout
allows it.

On disk, metadata is interleaved with bulk data: the .hpln payload is all buffer
bytes concatenated in write order with payload_offset assigned leaf by leaf, so a
column's small metadata channels sit either side of its multi-gigabyte packed
buffer. Reading just the metadata is one small scattered read per column per
channel.

In memory, pinned_entry::group_bounds is vector<vector<packed_column_bounds>> with
three owning vectors inside each -- roughly 3 * n_columns * n_chunks allocations,
chunk-major. Evaluating one column's filter is a pointer chase across the table,
and there is no single buffer to hand a kernel or copy H2D.

Proposed: one contiguous metadata region per table, column-major over groups, with
parallel typed arrays inside a column and a small directory giving per (column,
chunk) offsets. At SF1000 lineitem with 8192-row groups that is 11.7 MB contiguous
per column -- one sequential read, not 732k lookups.

Segregation buys more than tidiness: it is what makes read-metadata-then-fetch
possible at all, it lets the index stay device-resident while the payload spills,
it makes evaluation one H2D copy and one kernel launch, and it makes the region
independently cacheable over object storage.

For the file format, recommends a separate contiguous section with its directory
in the footer rather than a sidecar object: data and index stay atomically
consistent, which matters because a stale index is a correctness bug, and
byte-range caching already gives the independent-caching benefit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…n-major arena

The index only earns its keep if it can be read WITHOUT reading the data it
describes -- read metadata, decide, then fetch only what survived. The previous
shape could not: packed_column_bounds owned three vectors and the entry held
vector<vector<packed_column_bounds>>, roughly 3 * n_columns * n_chunks separate
allocations in chunk-major order. Evaluating one column's filter was a pointer
chase across the table, and there was no single buffer to hand a kernel, copy H2D,
or write as one range.

group_bounds_arena holds every (column, chunk, group) in one allocation,
column-major over (chunk, group), with mins and maxs back to back per slice.
packed_column_bounds becomes a non-owning view of a slice. A query filters on one
to three columns, so column-major makes those a few long sequential runs and never
touches the rest; chunk-major interleaved columns and forced a strided walk. It is
also the layout a GPU evaluator wants, so a device mirror is a straight copy.

At SF1000 lineitem with 8192-row groups this is ~11.7 MB contiguous per column.

from_capture refuses anything inconsistent -- differing column counts, differing
group_rows across chunks, a zero group_rows -- and returns an empty arena, which
means no sub-chunk pruning rather than a wrong one. Chunks may legitimately differ
in group count, since a pin's last chunk is usually short.

Writing the layout tests caught a heap overflow in the first version of the
packing: total_groups summed over chunks but not columns, so the arena
under-allocated by a factor of n_columns and writing the second column's bounds
ran past the end. Fixed, and the contiguity assertions now pin it.

valid is one byte per group rather than a bitset: it is read per cell on the hot
path, and a bitset would trade a byte per group (0.01% of the region) for a shift
and mask.

Evaluation stays 24x faster than the BaseStatistics path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…essed chunks

Wires the plan's survivor row ranges into cached serving. The cursor now walks
ranges instead of chunk indices when it is safe to, and an uncompressed device
chunk narrows its column VIEWS to the range while ownership stays the whole
column -- cudf slicing is a view operation, so rows are skipped with no copy and
no reallocation.

Two guards, both because something downstream assumes a batch is a whole chunk:

- Late materialization addresses a deferred row by its position in pinned-table
  order and decomposes it as local = gid - range.start, chunk = local / 1024. A
  mid-chunk slice shifts that decomposition, and column_origin.hpp warns the
  failure is a silent off-by-a-chunk rather than an error. So a scan whose origin
  annotation was stamped serves whole chunks and gives up sub-chunk pruning. This
  costs nothing where the pruning pays: the late-mat install gate needs
  pinned_column_null_count, which refuses any entry holding a compressed chunk, so
  a compressed pin is never stamped.
- An MVCC keep-mask is positional against the whole chunk, so a slice would
  misalign it. That chunk alone falls back to being served whole.

A range's end may exceed a chunk whose last group is short, so the slice clamps to
the column's real size rather than trusting the plan's arithmetic.

Compressed chunks still serve whole: skipping their rows means handing the decode
a row selection rather than slicing a materialized column, which is the next
piece and where the measured host-tier benefit lives.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…till need

Serving surviving row ranges works for uncompressed chunks. Compressed chunks
still serve whole, and this records exactly what closes the gap so it is not
re-derived.

Of the three enumerators that can express a selection, only chunk_csr actually
skips work -- mask_bits and index_list run a dense grid and merely compact the
output. But chunk_csr stores in_chunk_rows, a uint16 in-chunk position per
surviving row: ~190 MB of device memory for a 189M-row chunk at 50% selectivity,
to express a selection whose content is "all 1024 rows of these chunks", where
every uint16 is the sequence 0..1023 repeated.

What is needed is a dense chunk-list enumerator: block b decodes all of
chunk_ids[b] and writes at b * 1024. That is one uint32 per surviving chunk --
~370 KB instead of ~190 MB, 500x less -- and no per-row bookkeeping to build. It
fits the group index exactly, since groups are a whole number of 1024-row decode
chunks by construction, so a surviving range converts to a chunk-id run with a
shift.

Also records the order: the enumerator unlocks decode skipping for bitpack-rooted
columns and needs no format change; the group-to-byte table unlocks skipping the
transfer, which is where the measured host-tier benefit is. The two are
independent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…oncile the doc

Answers what the fetch path requires, and corrects a claim made one commit ago.

The key structural fact: bp_offsets -- where each 1024-row chunk's bits start
inside packed -- is not stored. It is computed at decode time by an exclusive scan
over the chunk_count and chunk_bits arrays that were loaded. So loading only the
surviving chunks' metadata entries and only their packed bytes, concatenated in
order, makes the decode-time scan produce correct offsets into the compacted
buffer, and an unmodified full decode of that smaller table yields exactly the
surviving rows.

That means the fetch path needs NO new decode enumerator -- the thing the previous
commit said compressed chunks need. Skipping transfer and skipping launches are
separate problems with separate solutions, and the transfer one is both more
valuable (-8.1% measured, against a ~2.4% envelope on a GPU-resident pin) and
cheaper. The enumerator section is rescoped to GPU-resident chunks accordingly.

Lists the pieces in dependency order and audits every earlier claim against what
is now known. One genuinely new requirement falls out that the plan did not
anticipate: an operator-level notion of "per-chunk metadata channel" versus "bulk
channel", which is what makes compaction expressible without special-casing
bitpack throughout.

Also reorders the document. Two sections had been appended out of position as 6A
and 7A; they are now 6.6 and 7.5 in place, and 7.5.5 records that the in-memory
arena it asked for is done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…hunks

Turns "these 1024-row chunks survive" into "these bytes to move", as pure
arithmetic over host-side metadata -- no GPU, no I/O, so it is testable directly.

plan_bitpack_packed_subset derives each chunk's word span the same way
simpatico_compute_bp_offsets_scan does on device, in one forward pass carrying the
prefix rather than a prefix sum per survivor. That equivalence is the whole point:
the compacted buffer's own scan must reproduce the boundaries these ranges were
cut on, or the decode reads neighbouring chunks' bits as values -- silently, not
as a failure. A test pins that all-chunks-surviving reproduces the full buffer
exactly, so the subset path degrades to the whole-buffer fetch rather than to
something subtly different.

Two details that are easy to get wrong and are now covered:
- The compacted size carries three decode guard words. simpatico_bitunpack_one
  loads packed[w .. w+2] unconditionally, so the last live word must stay
  readable two words past its end.
- A zero-bit chunk (every value equal to chunk_min) occupies no bytes but is
  still a surviving chunk: it must contribute no range and must not shift its
  neighbours.

append_coalesced merges touching ranges and can bridge a caller-set gap,
deliberately moving pruned bytes when a separate request would cost more than the
wasted bytes -- the policy knob the network path needs. The default of 0 merges
only adjacent ranges.

Nothing consumes this yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Records that where the byte ranges come from differs by backend while the result
type does not, so the fetch machinery, coalescing policy and header synthesis are
written once. A host pin derives them from the column's own per-chunk metadata
sitting in pinned host memory -- a memcpy, no round trip, and no format change at
all. Over a network those same 5 B/chunk arrays are scattered through the payload
region and cost a request each, which is what the stored group-to-byte table buys.

Disk and S3 stay the eventual target; the host path is the cheap way to prove the
mechanism, not a different mechanism.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Specifies the next piece of the fetch path concretely: signature, the two-phase
structure (read the small per-chunk metadata, then compute ranges and
re-lay-out), the per-buffer classification it dispatches on, and the bookkeeping
that has to move with it -- buffer size_bytes and num_rows, leaf num_rows, and
the column's num_rows, which are three different quantities since
leaf_desc::num_rows is the node's own output length rather than the column's.

The reader needs no change: it allocates each leaf at its declared size and fills
it through payload_fetch_fn, so a header declaring compacted sizes plus a
gathering fetch is enough.

Also records why this wants its own pass: it is binary-format rewriting where a
wrong size_bytes does not fault. The reader allocates what the header says and
the decode reads whatever landed there, so the failure mode is wrong values
rather than a crash -- the same hazard the byte-range arithmetic was tested
against, and it wants the same treatment: a test that every-chunk-surviving
reproduces the original header byte-for-byte, proving the subset path degrades to
the existing one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…t the seam

The byte-range arithmetic had bitpack in the caller's face: plan_bitpack_packed_subset
was the entry point, so any code wanting a subset had to know what plan the column
used. That is the same shape as deriving the min/max index from bitpack's channels,
and it invites the same failure -- a plan change silently taking a capability away.

Two changes.

Splits bulk_chunked, which conflated two very different things. A zigzag, identity
or str_split-offsets buffer is fixed bytes per ROW, so chunk c's extent follows from
the row count alone; only bitpack's packed is variable, sized by chunk_count x
chunk_bits. They are now bulk_fixed_stride and bulk_variable, which makes explicit
that exactly one buffer in the whole registry needs operator-specific arithmetic.

Adds plan_buffer_subset(kind, buffer, ...) as the entry point, dispatching on the
registered layout. per_chunk_metadata and bulk_fixed_stride are handled generically;
bulk_variable is the only branch that consults its operator, and it refuses without
the sizing metadata rather than guessing. whole_column, an unclassified operator and
an unknown buffer all refuse, which the caller must read as "fetch this whole" --
always correct, merely less selective.

So the layers separate cleanly: the min/max index is plan-agnostic and always was,
being computed from uncompressed values at pin time; addressing is generic for every
buffer but one; and the stored group-to-byte table, when it arrives for disk and S3,
makes even that one generic by supplying extents rather than deriving them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ntry

Closes the gap that made the sub-chunk pruning machinery inert: nothing called
compute_pinned_group_stats and nothing populated pinned_entry::group_bounds, so
the plan read a field no code ever wrote and survivor_row_ranges was always empty
in production.

Mirrors the coarse capture's plumbing. pin_materialization_options gains
group_rows; materialize_pin_batches runs the finer capture on the same table with
the same column types, gated behind capture_chunk_stats since the coarse pass is
the cheap first filter the fine one refines; host_pin_result and device_pin_result
carry it; insert_pinned_entry_host and insert_pinned_entry_device build the arena.
New setting pinned_zone_map_group_rows, default 8192 -- eight simpatico decode
chunks, where the measured curves flatten.

Verified on the clustered SF100 with a host pin: q6 drops 47.5M further rows
inside its 8 surviving chunks and q14 drops 32.1M inside its 2, on top of the
whole-chunk pruning that was already happening.

Two paths deliberately left out, both because they insert through the
merge-capable insert_pinned_entry, which appends columns to an existing entry:
group bounds would need the same merge and degradation handling
pinned_zone_maps::append_column_from provides, and getting that wrong on a merge
means bounds describing the wrong column. The uncompressed GPU pin path is the one
that loses out; it keeps whole-chunk pruning.

Also logs what sub-chunk pruning dropped, since a plan that silently narrows
nothing is otherwise indistinguishable from one that is not running.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

SF1000 clustered, host pin, 2GB batches, pruning on in both arms so the only
variable is the finer capture:

  pinned_zone_map_group_rows=8192  44.6e9 sub-chunk rows dropped  9.4813 s
  pinned_zone_map_group_rows=0                                0  9.6943 s

-2.20%, on top of the -8.1% whole-chunk pruning already measured. It moves the
scan-bound queries and nothing else: q12, q14, q8, q4, q15, q20 all improve; two
queries move the other way by less than the noise floor.

Confirms the prediction that a G=8 index sees through the row-group interleaving
the scan's coalescer introduces, without any pin-time clustering.

Records what it is not: these chunks are still fetched whole, so this is
downstream saving rather than transfer saving. 44.6e9 rows dropped for 0.21 s says
the same thing the ceiling analysis did -- rows are cheap once resident, and the
value of skipping is dominated by what it lets you not move. The fetch skip
remains the larger prize.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ed in place

Reading the writer shows the change is smaller than the design assumed.
leaf_buffer_desc::num_rows is not serialized -- the writer emits only name,
type_tag, size_bytes and payload_offset per buffer, and num_rows is derived on
read. So exactly four numeric fields move: a buffer's size_bytes and
payload_offset, the leaf's num_rows, and the column's num_rows. The last two are
different quantities, since leaf_desc::num_rows is the node's own output length
rather than the column's.

Recommends patching those fields in place rather than re-emitting from parsed
records. All four are fixed-width at computable offsets, so overwriting them keeps
the output structurally identical to what build_compressed_table_header produces,
whereas a second writer could drift from the real one and fail silently later.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ed table

Serving only some of a column's 1024-row chunks needs no read-side change:
bp_offsets -- where each chunk's bits start inside a bitpack `packed` buffer --
is not stored, it is scanned at decode time from the chunk_count/chunk_bits that
were loaded. Load only the surviving chunks' metadata entries and only their
packed bytes, concatenated in order, and that same scan produces correct offsets
into the compacted buffer, so an ordinary full decode of the smaller table yields
exactly the surviving rows.

read_compressed_table_from_memory already allocates each leaf buffer at its
declared size and fills it through a fetch callback, so all that is needed is a
header whose declared sizes are the compacted ones plus a fetch that gathers the
right ranges. build_chunk_subset_header produces both.

Four numeric fields change: a buffer's size_bytes and payload_offset, the leaf's
num_rows (the node's own output length, which drives the decode grid) and the
column's num_rows (whose last chunk is short). They are PATCHED in a copy of the
original header rather than re-emitted from the parsed records: every one is
fixed-width at an offset the parser can record, and a second writer would be free
to drift from build_compressed_table_header, with the divergence surfacing only
as wrong query results. parse_hpln_header gained an optional out-parameter that
records those offsets.

A column that cannot be subsetted -- a whole_column buffer such as a snappy root,
an unclassified operator, a node whose own length differs from the column's (so
column chunk ids do not name its rows), or a bitpack leaf whose sizing metadata
could not be read -- is emitted whole. Fetching whole is always correct, merely
less selective, so a mixed table is served rather than refused.

Each buffer is additionally checked against itself: with every chunk surviving,
the layout model must reproduce the size the writer declared, otherwise our idea
of that buffer's layout disagrees with what was written and a subset built on it
would be wrong without faulting. That check is what makes the all-surviving case
reproduce the original header byte for byte, which the new test asserts alongside
round-trips comparing decoded values to the surviving rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…header work found

Opens the document with where things actually stand: what is measured (host-tier
best config 10.44 s -> 9.48 s at SF1000, from whole-chunk pruning plus the
per-group index), what runs end to end, what is built but not yet connected, and
the known gaps in the order they will bite. A reader arriving cold should not have
to reconstruct that from a hundred pages of reasoning.

States plainly that the compressed fetch path is inert: three pieces exist with
host tests and nothing calls them. That is worth flagging loudly, because inert
machinery passing its unit tests is exactly how the per-group index sat unused
until someone checked for callers.

Also records the two requirements the header implementation turned up that the
design had not anticipated, both of which would have produced silently wrong data:
a leaf whose num_rows differs from its column's chunks on a different grid, so
survivor ids do not name the same rows there; and a per-buffer self-check that
every-chunk-surviving reproduces the declared size, which makes the byte-for-byte
property hold by construction and caught a latent zero-width-chunk divergence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…he serve shape

Two changes that could not be separated: wiring range-skipped fetch into the
host-tier compressed serve path forced the provider to decide what shape each
chunk's batches take, and that is where a shipped bug was.

**The bug.** Serving `survivor_row_ranges` walked one batch per range, but only ONE
of the four serve paths narrowed anything: a device_pin_chunk's uncompressed columns.
Per-column device storage, a compressed chunk (either tier) and an uncompressed HOST
chunk all ignored the range and served the WHOLE chunk -- once per range. A chunk
with two surviving groups was emitted twice. TPC-H q5 on clustered SF100, host tier,
pinned_zone_map_group_rows=8192 returns revenue ~6% high; correct at group_rows=0.

The provider now lays out its batches up front and asks, per chunk, what that chunk
can serve: an uncompressed device chunk takes one batch per range (cudf::slice, now
on the per-column path too); everything else takes one batch carrying all its ranges.
A chunk with an MVCC keep-mask still serves whole.

**The feature.** A host-tier compressed chunk turns its ranges into the surviving
1024-row decode chunks, and decompress_host_to_gpu synthesizes a header describing
only those (build_chunk_subset_header) and gathers only their bytes out of the pinned
payload. The reader and the decode see an ordinary, smaller table. The projection
reports the rows it will produce and scales its byte footprints, so a reservation
sized off it fits.

build_chunk_subset_header now reports, per column, whether it was compacted. It has
to: a column the format cannot address per chunk is emitted whole, and a whole column
beside a compacted one has a different row count -- `cudf::table` rejects the pair,
the query falls back to DuckDB, and the measurement reads +48% while still validating
22/22. The subset is used only when every column the scan reads was compacted.

SF1000 / host / 2 GB: 9.6845 -> 9.5848 s (-1.03%) against the same build with
SIRIUS_EXP_CHUNK_SUBSET_FETCH=0. 22/22 byte-exact on clustered SF100 at both tiers.
The win is capped by string columns, not by the mechanism: a dictionary- or
str_split-rooted column is not chunk-addressable, which is why q1/q4/q12 refuse and
q6/q14 engage. CHUNK_SKIPPING_PLAN.md 6.5.4/6.5.5 records both findings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… from column state

A dictionary-encoded string column refused to be served as a subset of its chunks
because its keys are a whole_column buffer, and one of those refuses the column. That
conflated two different facts:

  an lz4 payload   fetched whole, and its bytes depend on WHICH rows are served
  a dictionary key fetched whole, and its bytes do NOT

Only the first is a reason to refuse. ChannelLayout gains `column_state` for the
second, supports_chunk_subset accepts it, and the builder emits such a buffer whole --
num_rows and size_bytes untouched, only its payload offset moved -- while the indices
compact on the column's 1024-row grid. An ordinary decode then gathers the surviving
indices against the full keys.

The mark has to follow the TREE, not just the buffer. When the keys are themselves
compressed (`dictionary.keys_offsets -> bitpack`, as o_orderpriority and
l_shipinstruct do) that leaf's buffers look row-indexed on their own grid and its
num_rows is the key count, which the old code read as a refusal. So the builder walks
the edges from the root and marks every node reached through a column-state channel.
o_clerk's `keys_chars -> ans` and `keys_offsets -> delta -> rle` fall out for free.

`null_mask` is deliberately left unclassified: it is one BIT per row, which no layout
describes, so a nullable dictionary column refuses rather than being addressed with
byte-per-row arithmetic.

q1 and q4 now engage; SF1000 / host / 2 GB goes to -1.33% (9.6680 -> 9.5395 s, pooled
over two independent A/B pairs), 22/22 byte-exact on clustered SF100. Only str_split
columns (l_shipmode, q12) still force a whole-chunk serve; compacting those means
rebasing cumulative offset VALUES rather than gathering byte ranges, which needs its
own design (CHUNK_SKIPPING_PLAN.md 6.5.6).

Tests cover the three shapes TPC-H uses -- bare dictionary, bitpacked indices,
bitpacked keys and indices -- each asserting that every chunk surviving reproduces the
original header byte for byte and that a subset decodes to exactly the surviving rows.
NB the simpatico host tests are not built by `pixi run make`; build `--target all`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…thing to prune

Every pruning number in this project so far came from a pre-sorted dataset. TPC-H
as generated prunes 0.00%, and sorting the FILES does not fix it either: the scan's
row-group coalescer interleaves row groups, so a pin chunk spans the whole key range
even when the file is globally sorted (CHUNK_SKIPPING_PLAN.md 3.8). The clustering has
to happen after coalescing -- which means at pin time.

`CALL pin_table(..., cluster_by=['l_shipdate'])` sorts each chunk in
materialize_pin_batches, immediately after materialization and before anything observes
it: the zone maps must describe the order the rows are stored in, and every id handed
out downstream is positional against that order.

UNSORTED SF1000, host tier, 2 GB batches, best-of-3:

  no clustering   9.7569 s
  cluster_by      8.9210 s      -8.57%

for +1.6% pin cost (127 s -> 129 s whole-process wall), against the ~1.5% predicted in
5.1. The scan-bound queries roughly halve: q6 0.2350 -> 0.1183, q15 0.2562 -> 0.1293,
q14 0.2590 -> 0.1491. 22/22 byte-exact at SF100.

This confirms 5.2's central claim end to end: a LOCAL sort (per chunk, no shuffle)
prunes nothing at chunk granularity and everything through the per-group index. At SF10
a one-month predicate drops 59.1M of 60.0M rows in 19 row ranges over 19 surviving
chunks -- one contiguous run per chunk, zero chunks pruned coarsely -- and drops nothing
at all without cluster_by. It also beats the pre-sorted dataset (8.92 s vs 9.54 s),
since sorting after the coalescer is strictly better than sorting the files before it.

Refused for duckdb-native pins: their rows stay addressable by DuckDB row id and the
deleted-row keep-masks are positional against the pinned order, so reordering would
misapply them silently. An unpinned or unknown key column is an error rather than a
silent no-op, and cluster_by without a group index warns (a per-chunk sort alone prunes
nothing).

One query really regresses: q12, +0.03 s (~7% of q12), reproduced across two runs.
Choosing the key without being told is the open follow-up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…lans

Closes the "re-explore the plans against the clustered layout" follow-up as a
NEGATIVE result, which is worth recording because "those plans were tuned on
unclustered data" is the obvious objection to clustering.

Measured on one 16.8M-row SF1000 pin chunk, committed plans, file order vs sorted
by the cluster key:

  l_orderkey   delta -> bitpack   12.393x -> 2.739x    10.8 MB -> 49.0 MB
  o_orderkey   delta -> bitpack   12.393x -> 2.521x    10.8 MB -> 53.2 MB
  l_shipdate   bitpack             2.651x -> 142.932x  25.3 MB ->  0.5 MB
  o_orderstatus dictionary->bitpack 19.321x -> 235.044x 4.3 MB ->  0.4 MB

Every other column is unchanged to three decimals -- only columns correlated with
the sort key move. Net: lineitem -2.4%, orders +1.2%, so clustering is roughly
footprint-neutral and just moves bytes from the date columns to the order keys.

Re-exploring (--score pareto --rerank-top 16) returns plain bitpack for both keys at
2.706x/2.496x: the loss is intrinsic, since sorting on shipdate scatters the order key
and no cascade compresses a scattered 64-bit key. The one thing it does find is that
the delta is now dead weight -- 0.1% more bytes, decode 1126 -> 1435 GB/s. That does
not reach the suite: 8.9547 s with the re-picked plans vs 8.8539 s with the committed
ones (+1.14%, inside the ~0.8% noise floor). Keep the committed plans.

Also corrects 5.4: q17's +0.080 s was noise (a same-config repeat gives +0.002 s).
q12 is the one real regression, at +0.027/+0.038 s across two runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ans lose

Two measured results, both of which correct something the project believed.

1. THE LADDER (3.13). Every mechanism added one at a time on UNSORTED SF1000, all
eight tables host-pinned, 2 GB batches, best-of-3:

  none      9.6544
  coarse    9.7090   +0.56%   whole-chunk zone maps
  group     9.7179   +0.09%   + per-group index
  subset    9.7885   +0.73%   + range-skipped fetch
  cluster   8.7972  -10.13%   + pin-time clustering
  allplans  8.7739   -0.27%   + compression for the other four tables

Without clustering the pruning machinery is a NET LOSS of +1.39% -- it prunes 0.00%,
so it is pure overhead. Clustering is the switch that turns it on. Holding clustering
fixed and removing the rest attributes it: clustering alone -2.2%, the per-group index
+0.2% (it locates surviving rows but moves nothing), the fetch skip -7.0%. All three
are required and none pays alone. This supersedes 3.12's "-2.20% for the group index",
measured on the pre-sorted dataset with the 6.5.5 duplicate-rows bug live.

2. PLAN SELECTION (3.14). A host-tier cost model -- minimise 1/(ratio*370 GB/s) +
1/decode -- says decode throughput should dominate ratio, and scores the max-ratio pick
for p_partkey (193x at 136 GB/s) as 2.8x WORSE than not compressing. It predicted its
picks would be 34-40% cheaper per table. Measured end to end, mean of two runs:

  explored, max ratio with decode >= 250 GB/s   8.8211
  the *_disabled.txt originals                  8.9089  +1.00%
  cost-model, throughput-weighted               9.0023  +2.05%

The model's serial assumption is the error: the decode hides behind 18 scan threads
while the host->device copy is a shared resource on the critical path. So the committed
"max ratio with decode >= 250 GB/s" floor is closer to right than a throughput-weighted
rule. Recorded because the throughput argument is intuitively compelling and wrong.

Also settles the *_disabled.txt question: enabling those four tables' plans is worth
-0.27% against leaving them uncompressed, i.e. nothing measurable. No case for
re-enabling them on this workload.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Settles 7.2, which has gated the simpatico-ingestion line since the project opened:
does S3's internal striping mean a pruned ranged GET costs the backend the same as an
unpruned one? Measured on a g7e.2xlarge in us-east-2b against a bucket in us-east-2
(same region, so this is S3 and not an inter-region link), 4 GB object, stdlib HTTPS
ranged GETs with keep-alive.

Reading less takes proportionally less time -- 50% of an object in 0.500x the time,
25% in 0.259x. Fragmentation at equal volume costs 6x from 16 MB ranges down to 64 KB,
but entirely through REQUEST COUNT: throughput = min(NIC cap, concurrency * range /
RTT) predicts every point to within 4-23% with no term for the number of distinct
extents, and p50 latency FALLS as ranges shrink (254ms -> 24ms), so small ranges pay a
fixed ~25 ms floor rather than a per-byte penalty. The bytes you skip are free; what
you pay for is asking.

That also turns 7.3's coalescing policy from a TODO into a number. One extra request
amortises to 200-460 KB of transfer at concurrency 64 (three independent pairs), so:
bridge gaps under ~256 KB, coalesce to ~4-16 MB runs, and keep concurrency * range >=
cap * RTT or the pipe starves however well the ranges are merged. It also says a G=8
group (32 KB for a 4-byte column) is far too fine to address individually over S3 while
being exactly the right granularity to DECIDE with -- the index picks rows, the
coalescer picks requests.

Our fetch skip already produces runs far above that knee: 6.5 measured roughly one
contiguous run per buffer covering ~32% of it, tens of MB at a 512 MB batch. So
range-skipped ingestion from S3 would pay close to its byte fraction, for the same
reason everything else in this project does -- clustering is what turns scattered
survivors into long runs.

Caveats recorded in 7.6: throughput plateaus at ~0.99 GB/s, which is the instance NIC
rather than S3, so the large-range rows all sit at the cap; and the probe's fixed byte
budget starved concurrency at 64 MB, fixed here by holding request count constant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

a510f40 reported a 0.99 GB/s plateau and attributed it to the instance NIC. It was
the GIL: the python probe copies every response body through the interpreter. curl
--parallel on the same instance, same object, same ranges, reaches 2.85 GB/s
(22.8 Gb/s), saturating at 64-way -- 2.9x the python ceiling. A ceiling that ignores
added concurrency (64 and 128 threads identical) is the signature, and it looks exactly
like a network cap.

The correction matters because the coalescing policy scales with it. Re-derived from
curl at equal volume and equal parallelism, 4 MB ranges against 16 MB: 0.755 ms per
extra request at 2.6 GB/s = ~2 MB worth bridging, against the ~256 KB the
interpreter-bound numbers implied. A 7x error in a constant that would have been
hardcoded once and never revisited.

So 7.7 now states the policy as a formula -- max_gap_bytes ~ per_request_cost x
achievable_bandwidth, both measurable at runtime -- rather than a number, since it
moves with the deployment.

22.8 Gb/s is still ~2x under the instance's rated 50 Gb/s, so it is recorded as a floor:
the box may have been busy, and a single object is a single S3 prefix, which is the
classic way to leave object-store bandwidth unclaimed. A real scan spreads across many
files.

Unchanged: fragmentation costs request count, not skipped bytes. That rests on p50
latency FALLING as ranges shrink and on the model needing no extents term -- latency
facts, independent of the bandwidth ceiling -- and curl reproduces the same shape at
2.9x the bandwidth (4 MB costs +36% against 16 MB at equal volume).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…est cost is not

Repeat on an idle instance. The ceiling does not move: 2.65 / 2.70 / 2.51 / 2.63 GB/s
at 32 / 64 / 128 / 256-way against 2.21 / 2.84 / 2.85 / 2.63 busy, saturating by
32-way. So contention was not the limit and ~21 Gb/s is this path's real behaviour;
the ~2.4x gap to the instance's rated 50 Gb/s is structural, with one-object-is-
one-prefix the prime suspect.

The per-request cost, however, is load-sensitive: 0.286 ms idle against 0.755 ms busy,
so max_gap_bytes lands at 0.75 MB rather than the 1.99 MB the busy run implied. Taking
the idle row as the estimate and the spread as the error bar, the constant spans 7x
across the three ways it has now been derived (293 KB interpreter-bound, 0.75 MB idle,
1.99 MB busy) -- which is the argument for 7.7 stating it as a formula.

Also worth recording: the fragmentation penalty at 4 MB is modest, +13% idle against
16 MB at equal volume, not the +36% the busy run showed. It is the sub-MB ranges that
collapse, not the merely-smallish ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

First piece of file ingestion. An ingest path needs the schema and the byte budget
before it moves any payload -- which columns, how many rows, how much to allocate --
and today nothing can answer that: every read path takes a payload_fetch_fn and returns
a device-resident table, and the file reader pulls the WHOLE file into host memory
before parsing. That is the opposite of what a reader wants when the object is large or
remote.

describe_compressed_table_header() parses only the structural header: per column the
name, dtype tag, decimal scale, row count and compressed footprint, plus the header
length and the payload extent. No payload is touched and no GPU is involved. The
internals already computed all of it; it was simply never exposed.

It accepts a PREFIX of a larger buffer and ignores what follows, because that is how a
remote reader has to work -- and a truncated prefix is reported as an error rather than
partially parsed, which is how the caller learns to re-read with more bytes.

That guesswork is a real format gap, not a quirk of the API: .hpln carries no length
prefix and no footer, so the end of the header cannot be located without parsing it,
and it cannot be parsed without already having the bytes. CHUNK_SKIPPING_PLAN.md 7.5
argues for a segregated, self-locating metadata region; this is the first place the
absence actually costs something.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Second piece of file ingestion, and the one that makes it worth doing on its own.

A pin today reads parquet, materializes it on the GPU and compresses it on the way into
the cache. That is why pinning dominates the SF1000 wall clock -- ~151 s of pin to
serve ~9 s of queries. A file that is ALREADY in the pinned representation needs none
of it: read_hpln_into_pinned() stages the payload into pinned host blocks byte-for-byte
as stored. Nothing is decoded and no GPU is touched, so ingest is an I/O copy rather
than a decode plus a re-compress.

The serve side needs no new code at all. What lands is an ordinary
compressed_host_representation over an ordinary pinned_compressed_blob, so it goes
through the same converter as a pinned chunk -- including the range-skipped fetch from
6.5. That was the argument for staging into the pinned form rather than inventing an
ingest-specific path, and the test asserts it by serving through
converter_registry::convert rather than through anything ingest-specific.

Locating the header is the awkward part and it is the format's fault: .hpln has no
length prefix and no footer, so the header's extent cannot be known without parsing it.
The reader grows a speculative prefix (1 MiB, doubling to 64 MiB) and only treats a
parse failure as fatal once the prefix covers the file, since until then "malformed"
and "not enough bytes yet" are indistinguishable. Over a network that guesswork becomes
an extra round trip, which is a concrete argument for 7.5's self-locating metadata
region.

Tested by writing a file, ingesting it, and comparing DECODED VALUES -- a wrong payload
offset does not fault, it returns neighbouring bytes as values. Verified the assertion
has teeth by shifting the payload read by one byte and confirming the test fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…egment table

The layout was [header][payload] with nothing recording where the header ends, so a
reader could not locate anything without parsing the header and could not parse the
header without already holding it. read_hpln_into_pinned had to guess a prefix and
re-read when it guessed short -- locally a wasted read, over a network a round trip on
every open.

A file now ends with a fixed 16-byte trailer (postscript offset, postscript length,
version, magic 'HPLN' last so a reader validates from the tail) pointing at a postscript
that locates segments:

    [header][payload][segment...][postscript][trailer]

Read the last 64 KB and you know where everything is. When that tail is too short the
reader is told exactly how many bytes WOULD do, so the re-read is exact rather than a
doubling search.

Segments are keyed by kind and a reader skips kinds it does not know, so this is the
last format break needed for a while: zone maps land next, null masks are in flight on
another branch, and 6.1's group->byte table will follow -- none of them need to disturb
this. That extensibility is the point, and it is the same shape Vortex uses (trailer ->
postscript -> DType / Layout / Statistics locators) for the same reasons.

The in-memory header path is deliberately untouched: pins build headers with
build_compressed_table_header and the chunk-subset code patches them in place, so the
FILE wraps the header rather than changing it. read_compressed_table still parses from
the front and still works.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…arry

An ingested file has to prune without decoding anything, which means the bounds must
travel WITH the data rather than being recomputed from it. group_bounds_arena gains
pack()/unpack() producing the engine-neutral bytes the `zone_maps` segment carries: per
column a type tag and two null flags, then per (column, chunk) the group count followed
by mins, maxs and validity. The layout mirrors the arena's own, so decoding is a copy
rather than a rebuild -- which matters because this sits between a file open and the
first query.

Types are packed as a small stable tag set covering exactly what
compute_pinned_group_stats can capture (integers <= 8 B, DATE, TIMESTAMP). Anything else
packs as "no statistics", and a malformed segment unpacks to an EMPTY arena rather than
throwing, so both degrade to serving the file unpruned -- never to pruning wrongly.

The test asserts the property that actually matters: not that the bytes round-trip, but
that the decoded arena makes the SAME pruning decisions, compared through
lowered_bound_filter::select_survivors. A bound that survives serialisation shifted by
one is a silently wrong answer rather than a crash, so I verified the assertion has
teeth by offsetting the packed mins by one element and confirming the test fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…able prunes

Joins the two halves. write_table_to_hpln() compresses a table and writes it with a
zone_maps segment; read_hpln_into_pinned() locates that segment through the postscript
and unpacks it into the ingested table's group_bounds. An ingested file now prunes
WITHOUT decoding anything.

A writer that emits statistics is not optional here: once a file holds compressed bytes
the bounds are gone until someone decodes them, which is exactly the work pruning exists
to avoid. So the statistics are computed from the decoded values at write time, on the
way past, and travel with the data.

Every failure degrades to serving unpruned rather than pruning wrongly -- a file written
before the segment existed, a short read of it, or a segment that does not decode all
leave group_bounds empty and log rather than throw.

Tested end to end on an ascending column: export at group_rows=1024, ingest, and check
the bounds recovered from the FILE (mins 0/1024/2048..., maxs 1023/2047/...) prune
`v < 2500` to exactly groups {0,1,2}. Nothing in that test decompresses, which is the
claim. Verified the assertion has teeth by dropping the segment at write time and
confirming the test fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…le-endian zone maps

Two gaps closed.

LOGICAL TYPES. The header carries cuDF physical types, which cannot express DECIMAL
precision (cuDF tracks scale only, so DECIMAL(12,2) and DECIMAL(18,2) are
indistinguishable once compressed), nullability, or a timestamp's time zone. A pin gets
those from memory alongside the data; a FILE has nowhere else to get them, so it was not
self-describing. A `logical_types` segment now travels with every written file,
positional with the header's columns, using OUR stable tag numbering rather than
duckdb::LogicalTypeId's so a DuckDB upgrade that renumbers its enum cannot silently
reinterpret existing files. An unrepresentable type packs as SQLNULL -- "no declared
type" -- rather than a wrong substitute.

ENDIANNESS. The zone-map packing used a raw memcpy of the host representation while the
rest of the format uses explicit little-endian (push_le/read_le). Both agree on every
target this runs on, so it was convention debt rather than a live bug -- but it would
have disagreed by producing wrong BOUNDS, which prune wrong rows rather than failing.
Now explicitly little-endian like everything else.

Also tests the pre-trailer fallback for the first time. read_hpln_into_pinned grows a
speculative prefix for files written before the trailer existed, and that path had no
coverage -- which is how a fallback quietly stops working. The test writes the old
[header][payload] shape by hand and checks it ingests, with no statistics and no
declared types, serving unpruned rather than failing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Scopes the work against op::scan::gpu_ingestible, which is the interface every source
implements and therefore says exactly what is missing rather than what seems missing.
Roughly half is already built -- bind-time schema, per-group statistics (better than
parquet's row-group granularity), and range-skipped fetch -- and the missing half is the
metadata walk, the coalescer, a transport and the SQL surface.

Two conclusions worth recording because they change the plan:

The "identity segment" sketched earlier is DROPPED. It existed so a pinned .hpln could be
matched to queries over the parquet dataset it caches; a real read_simpatico() source
makes cache identity just the file path, exactly as for parquet. It was a workaround for
not having a source, and it is more total work.

One file is one compressed_table, and that blocks the INGESTIBLE rather than just the
pin: with one chunk there is nothing for a metadata walk to emit. Prefer a chunk
directory segment over one-file-per-chunk, so a table stays one object and the trailer's
single tail read keeps paying off over S3.

Milestones A-F, with the note to do multi-chunk before pruning: chunking decides what a
surviving chunk means in a file, so pruning first means designing it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…s, no payload

First piece of milestone A (7.9). read_hpln_schema() opens a file far enough to answer
"what columns does this have" and no further: names, the engine's logical types, and the
row count, with no payload staged and no GPU work. Binding a query must not move data.

Every downstream consumer asks exactly this question -- a read_simpatico() bind, an
ingestible's table_info(), and a pin over a file -- so it is the piece to build first.

Types come from the file's logical_types segment when present. When it is absent the
schema is derived from the cuDF physical types instead, which is lossy in the ways 7.9
lists: a DATE comes back as INTEGER, and DECIMAL precision becomes the carrier's maximum
because cuDF records only scale. A file without declared types binds approximately
rather than refusing, and the test asserts the degradation explicitly rather than
pretending it does not happen.

Works on both file shapes: located through the trailer when there is one, and by growing
a header prefix from the front when there is not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A .hpln has been readable as a pinned chunk for a while, but a pin is not a
source: the scan operator drives a metadata walk that emits splits, a coalescer
that batches them, and a materialize that turns a batch into a GPU table. This
adds that shape over a .hpln file.

The file holds one compressed chunk, so the walk emits exactly one split and the
coalescer passes it straight through -- both are still written as the general
contract rather than special-cased away. There is no projection and no row
filter, so post_filter_and_project only drops the positions the caller says it
will overwrite, and every optional virtual keeps its conservative default:
has_row_filter and can_report_survivors stay false, so nothing installs a late
materialization over this source and discovers at runtime that it cannot
report survivors.

The decode is the existing pinned-chunk path: read_hpln_into_pinned stages the
payload byte-for-byte and read_compressed_table_from_memory fetches leaf buffers
out of those pinned blocks. It decodes on the caller's stream rather than on the
decode thread pool -- one chunk per file does not repay having to re-bind
pool-stream buffers before the pipeline may read them -- and synchronizes once
after the fetch, because a host free is not stream-ordered and the staging blob
dies with the call.

Sizing the split needed a piece the bind schema did not carry. The reservation
must match the DECODED cuDF table, and the declared logical type is not always
its physical layout: the first version estimated a DECIMAL(12,2) at 8 bytes a
row where the file decodes it at 4, a third over-reservation on that column.
read_hpln_schema now reports the cuDF type per column alongside the engine type.

Tested by driving the interface as the scan manager's driver loop does: bind,
walk to exhaustion, coalesce, materialize, post-filter. Three negative controls,
each confirmed to fail before being reverted -- shifting the payload fetch by one
byte (decoded values differ, no fault), dropping the split in the coalescer (the
walk yields nothing), and ignoring the elided positions (the projection keeps a
column it was told to drop).

Not yet: multi-chunk containers, zone-map pruning into a ranged fetch, a remote
transport (the ioctx resolver is accepted and unused; the ingest reads through
the filesystem), and any SQL surface. A variable-width column's reservation
counts only its offsets, so a file with strings under-reserves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A .hpln was readable only from C++: the gpu_ingestible existed but nothing in
the engine constructed one. This gives it a SQL surface. `read_simpatico(path)`
binds through read_hpln_schema -- header plus logical_types segment, no payload
and no GPU, so a bind costs two small reads -- and the plan generator turns the
resulting TABLE_SCAN into a GPU scan over simpatico_gpu_ingestible, mirroring
what sirius_read_parquet does, path in parameters[0] included.

Two things fall out of wiring it up that the interface alone did not say.

Column projection is not optional. The projection create_plan(LogicalGet&)
pushes above a no-pushdown scan references the scan's output by POSITION, so a
source emitting the file's full width mis-references every column: SUM(b) over a
three-column file silently sums a. The ingestible therefore takes a column_ids
selection, decodes exactly those columns in exactly that order, and sizes its
reservation the same way. Out-of-range indices are refused at construction
because simpatico::decompress does not bounds-check its selection -- it would
return a neighbouring column's buffers as data.

A column's declared type and the type it decodes to need not agree, and nothing
converts between them: an INT32 payload declared DECIMAL(12,2) would be read
under a DECIMAL64 carrier. Such a column is refused during plan generation
rather than reinterpreted, per column rather than per file.

There is no CPU reader for the format, so there is no fallback. A GPU failure is
replayed on the CPU, reaches read_simpatico's own execute callback and errors
there; the message says so and points at enable_duckdb_fallback = false, which
is the only way to see the GPU error that actually caused it.

Scope is milestone A of CHUNK_SKIPPING_PLAN.md 7.9: one file, one chunk, one
split. No pruning, no filter pushdown, no remote transport, no writer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A .hpln held exactly one compressed_table, so one file was one chunk was one
split. That capped a file at whatever fits in a single compression pass, and it
also blocked the scan model itself: the engine walks metadata, emits splits and
coalesces them into batches, and with one chunk there was nothing to walk.

A file now holds N independently compressed chunks located by a new
`chunk_directory` segment (kind 5 — unknown kinds are skipped, so this is
additive and older files still read). The layout is segregated the way §7.5 of
CHUNK_SKIPPING_PLAN.md argues for: every chunk's structural header first,
contiguously, then every chunk's payload. The naive [hdr][pay][hdr][pay] is
easier to write and costs a seek per chunk to read the metadata of, which is
the access pattern the trailer exists to avoid — over object storage it is
request count, not skipped bytes, that is charged (§7.6). The `header` and
`payload` segments still bound their whole regions, so a reader fetches one
range and subdivides it with the directory; the directory also records each
chunk's row count, because the ingestible has to size a split before it decodes
anything.

All of a file's chunks must describe the same table. That is validated when the
headers are parsed — at bind — rather than discovered mid-scan, where the
symptom is a batch that cannot be concatenated or, with matching widths, one
that can and is wrong.

On the scan side the cursor is now a chunk index rather than a claimed flag, so
the walk emits one split per chunk, and the coalescer accumulates chunks up to
the same byte budget every other source batches to instead of passing each
through alone. A batch decodes its chunks and concatenates them in chunk-id
order. The zone-map segment carries every chunk's bounds in chunk order, so
pruning (milestone C) is wiring rather than a format change.

read_hpln_into_pinned still serves a single chunk and now refuses a multi-chunk
file outright: it yields one blob, so serving one would silently drop every
chunk but the first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>